PR 1: wire cua-cli non-interactive paths onto CuaAgentHarness - #22
Conversation
PR 1 of the cua-cli → CuaAgentHarness migration plan
(docs/cua-cli-harness-migration.md). Wires the non-interactive
surface of cua-cli onto CuaAgentHarness + pi 0.79 while leaving the
interactive TUI on the legacy stack for now.
Engine:
- harness.ts assembles a CuaAgentHarness from a Kernel client + browser
+ jsonl Session + cua-cli skills + pi-coding-agent's createCodingTools
as extraTools.
- harness-browser.ts provisions Kernel browsers via the SDK directly
(drops the cua-translator browserSession wrapper on new paths).
- harness-sessions.ts wraps JsonlSessionRepo for list / find-latest /
resolve-by-ref and is tolerant of unknown files in the sessions root.
- harness-models.ts resolves -m flags through @onkernel/cua-ai's
listCuaModels / parseCuaModelRef catalog; default is openai:gpt-5.5.
- harness-skills.ts loads skills via pi 0.79 loadSkills.
- harness-named-sessions.ts re-implements the named-session CLI on the
SDK.
CLI:
- print.ts and action/harness-runner.ts drive the harness for --print
and one-shot action subcommands (open/click/type/press/observe/url/
screenshot/do), aborting at maxTurns via harness.abort.
- output/harness-jsonl.ts sources the documented event schema from
harness.subscribe and stamps a schema_version field.
- cli.ts dispatches `cua models`, `--print`, action subcommands, and
`cua session ...` to the new wiring through cli-harness.ts. The
interactive entry point still uses the legacy stack.
Tests + CI:
- vitest config + fixtures: registerApiProvider-based scripted-provider
driver and a fake Kernel client that stubs browsers.computer.batch /
captureScreenshot. Test suites cover --print text + jsonl envelope,
action exit codes (ok / not_found / error / screenshot), session
resolution (list, latest, prefix, legacy-tolerance), model-ref
parsing (default, ref pass-through, bare-id, ambiguity), and harness
assembly invariants (coding-tools assignable to extraTools, default
system prompt + skill block composition, first-prompt screenshot via
harness.prompt({ images })).
- CI gains a cli-unit job that runs the new tests on every PR.
Old and new dep trees coexist intentionally in this PR; @mariozechner/*
stays on the package while interactive remains on it. PR 2 rebuilds
the TUI on harness + pi-tui 0.79 and PR 3 deletes the legacy code.
rgarcia
left a comment
rgarcia
left a comment
There was a problem hiding this comment.
Reviewed against docs/cua-cli-harness-migration.md (PR 1 section). Ran the build and all three test suites locally: typecheck green, cua-cli 18/18, cua-agent 28/28, cua-ai 88/88. Confirmed the node dist/cli.js ERR_MODULE_NOT_FOUND reproduces with main's sources (pre-existing, fine to defer). No edits under packages/agent or packages/ai; scope is clean; the three open questions are answered and test-backed.
Overall the architecture matches the plan well (harness assembly, SDK provisioning, jsonl repo, pi skills, env auth, scripted-provider fixture through the real buildCuaHarness). But a handful of behavioral contracts the plan calls out as "preserved" are not, so this needs another pass.
Major
-
One-shot action subcommands now persist a session file per invocation.
runActionCommand→setupHarnessRuntime→resolveSessionfalls through tocreateSession(cli-harness.ts:305) whenever--no-sessionisn't passed. LegacyrunActionSubskipped the SessionManager entirely for actions without-s, and the top-level README documents exactly that ("One-shot action subcommands (without-s) also skip the transcript"). Besides the disk clutter, everycua click ...now changes what-c/--session latestresolves to. Suggested fix: in the action path, only resolve a persistent session whenflags.namedSession(or an explicit session flag) is set; use the in-memory session otherwise. -
Action runner drops the first-prompt screenshot. Legacy actions went through
promptWithScreenshot, attaching a screenshot to the first user message of a fresh transcript. The newrunActioncallsopts.harness.prompt(prompt)with no images (action/harness-runner.ts:102), and only yutori injects a screenshot at payload time. Socua click/type/observe/url/dorun the first turn blind for openai/anthropic/google/tzafon —observe's prompt even forbids taking actions, so the model can't recover by calling the screenshot tool without violating its instructions.print.tsgot this right (maybeInitialScreenshot); the action path needs the same treatment, which also matches the plan'sagent-prompt.tsrow ("first prompt of a fresh session attaches a screenshot ...harness.prompt(text, { images })"). -
cua session start --profile <name>regressed.runSessionSubcommandpasses the raw--profileselector asprofileId(cli-harness.ts:532) andstartNamedSessionputs it straight intoprofile: { id: ... }(harness-named-sessions.ts:127-129). Legacy resolved name-or-id (and auto-created missing names) viaprofileSelector. The newharness-browser.tsalready hasresolveProfileIdwith exactly the legacy semantics —startNamedSessionshould use it. As written, a profile name is sent as an id, and the raw selector is persisted intometa.profile_id(legacy stored the resolved id). -
--session <path>and named-session transcript resolution are exact-string, cwd-filtered matches.resolveSessionRef(harness-sessions.ts:68) and the named-session continuation lookup (cli-harness.ts:300) compare the input againstmetadata.pathfromrepo.list({ cwd }). Consequences: relative paths fail; an absolute path to a session created from another cwd fails; andcua -s <name> --printrun from a different directory than where the transcript was created silently starts a fresh session and clobberstranscript_path. Legacy attached to any given path regardless of cwd. Suggested fix:path.resolvethe input and match againstrepo.list()without the cwd filter (or load metadata directly from the file) for the path/transcript-path cases. The contract list explicitly includes--session <path|prefix|latest>.
Minor
-
jsonl schema drift beyond the documented change.
browser_created.profile_idis now always omitted (profile_id: undefined, output/harness-jsonl.ts) — legacy emitted the resolved profile id when--profilewas used;CuaBrowserHandlejust doesn't carry it. Alsosession_created.modelchanged value format from bare id (gpt-5.5) to ref (openai:gpt-5.5). Both may be acceptable under schema_version=1, but the plan says "Note schema version in the README" and neither README nor the top-level "Session transcripts"/output docs were touched. -
Provider
*_BASE_URLenv overrides aren't wired on the new paths. The plan's config.ts row calls for "<PROVIDER>_BASE_URLenv overrides spread onto the model object (a few lines)", andcua --helpstill documents them. OnlyKERNEL_BASE_URLis honored. (Legacy applied at leastYUTORI_BASE_URLviaapplyProviderBaseUrl.) -
--thinkingsilently coerces unknown values tolow(mapThinkingLeveldefault case). A typo like--thinking hgihshould be a usage error (exit 2) per the exit-code contract, not silently low. -
runActionlost two legacy fallbacks: (a) when notext_deltaevents arrive, legacy pulled the final assistant message text from state before parsing — the new runner only accumulates deltas even thoughharness.prompt()returns the finalAssistantMessage; (b) tool-error extraction no longer prefersdetails.errorover content text. -
-c/latestsemantics changed from last-modified to last-created. Legacy sorted by file mtime;findLatestSessionsorts bycreatedAt(andrepo.listalready returns createdAt-desc, so the extra sort is redundant). After resuming an older session,-cnow continues a different session than legacy would. -
Test coverage gaps vs the acceptance list: no test for exit code 2 (error path), none for the
maxTurnsturn-cap abort, and the jsonl test has no tool steps sotool_call/tool_resultshapes are unasserted. All cheap to add on the existing fixtures.
Nit
HarnessRunOptions.session(andverbose) are required but unused inrunAction;cli-harnesseven builds a throwawayfallbackInMemorySession()to satisfy it, duplicating the in-memory session already created insetupHarnessRuntime. LetHarnessRuntimecarry the session it actually built the harness with, and drop the dead params.await import("@onkernel/cua-agent")forInMemorySessionRepoin two places in cli-harness.ts — the module is already statically imported at the top of the file.resolveAuthcould use cua-ai'srequireCuaEnvApiKey, which names the env vars to set, instead of the genericmissing API key for provider "x".buildCuaHarness'ssystemPromptcallback closes over the build-timeskillsarray; the callback receivesresources— usingresources.skillsmatches the plan wording and will matter once PR 2 callssetResources.- Transitional interop worth a PR-body note:
-s <name>transcript_pathis shared between the legacy interactive stack and the new repo format; a--print/action run rewrites it to a v2 path that the legacy interactive stack will then try to open. vitest.config.tssetsserver.host— a Vite dev-server option that does nothing undervitest --run.
Items 1-4 are should-fix-before-merge; the rest are take-or-leave. The structure is right and PR 2/3 should slot in cleanly once the action-path contracts are restored.
- Action subcommands no longer create on-disk session files unless an
explicit session flag is set (-s / -c / -r / --session).
- Action runner reattaches the legacy first-prompt screenshot via
harness.prompt({ images }) on fresh sessions.
- runAction falls back to the returned AssistantMessage text when no
text_delta events arrived, and prefers details.error over content
text when extracting tool errors.
- cua session start --profile <name> now goes through resolveProfileId
so a name is created/looked up before provisioning, and the resolved
id is what gets persisted in named-session metadata.
- --session <path> and named transcript_path resolution accept paths
from any cwd by reading the session header directly.
- -c / latest sorts by file mtime (legacy semantics) instead of header
createdAt.
- jsonl browser_created emits profile_id when --profile is used, and
the README documents the schema_version + model-ref change.
- <PROVIDER>_BASE_URL env overrides flow through buildCuaHarness onto
the resolved model object.
- --thinking values are validated up front; unknown values exit 2.
- HarnessRuntime exposes the assembled Session, dropping the
fallback-in-memory session and dynamic InMemorySessionRepo imports.
- systemPrompt callback reads resources.skills, so a future
setResources() call picks up the new skill set.
- resolveAuth uses requireCuaEnvApiKey for an env-var-named error.
- Tests cover error exit 2, the turn-cap abort path, jsonl tool steps,
the first-prompt screenshot, and --session <path> from a different
cwd.
|
Firetiger deploy monitoring skipped This PR didn't match the auto-monitor filter configured on your GitHub connection:
Reason: This PR is in the To monitor this PR anyway, reply with |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Skill print skips screenshot
runPrintnow captures first-turn images before dispatch and, when needed for/skill:prompts, sends a skill-formatted prompt with those images so fresh skill runs retain screenshot context.
- ✅ Fixed: Unknown skill slash command mishandled
- Unknown
/skill:<name>inputs are now expanded into the legacy missing-skill explanatory prompt plus remainder text instead of forwarding the raw slash command.
- Unknown
Or push these changes by commenting:
@cursor push 311cbe4cb1
You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit c4d78e9. Configure here.
| assistant = await opts.harness.skill(invocation.skill.name, invocation.remainder || undefined); | ||
| } else { | ||
| const images = await maybeInitialScreenshot(opts); | ||
| assistant = await opts.harness.prompt(opts.prompt, images ? { images } : undefined); |
There was a problem hiding this comment.
Skill print skips screenshot
Medium Severity
For --print, prompts that match /skill:<name> call harness.skill and never run maybeInitialScreenshot. On a new session (not resumed), the legacy path still attached a first-turn browser screenshot via promptWithScreenshot, which computer-use tasks often need.
Reviewed by Cursor Bugbot for commit c4d78e9. Configure here.
| assistant = await opts.harness.skill(invocation.skill.name, invocation.remainder || undefined); | ||
| } else { | ||
| const images = await maybeInitialScreenshot(opts); | ||
| assistant = await opts.harness.prompt(opts.prompt, images ? { images } : undefined); |
There was a problem hiding this comment.
Unknown skill slash command mishandled
Medium Severity
When --print receives /skill:<name> but no loaded skill matches, the harness path sends the raw slash command to harness.prompt. The previous stack expanded that into a clear “skill not found” message plus the user’s remainder text.
Reviewed by Cursor Bugbot for commit c4d78e9. Configure here.
1 participant
Summary
PR 1 of 4 in the
cua-cli→CuaAgentHarnessmigration plan (seedocs/cua-cli-harness-migration.md).Routes the non-interactive surface of
cua-cliontoCuaAgentHarness+pi 0.79 while leaving the interactive TUI on the legacy stack. Per the
plan, old and new dep trees coexist temporarily.
What changed
Engine (new wiring)
src/harness.ts— assembly. Builds aCuaAgentHarnessfrom a Kernelclient + browser +
Session+ skills + extraTools. UsesNodeExecutionEnv,createCodingTools(cwd)asextraTools, and asystemPromptcallback that composesresolveCuaRuntimeSpec(model).defaultSystemPromptwithformatSkillsForSystemPrompt(skills). Env-var API-key resolutionthrough cua-ai's
getCuaEnvApiKey.src/harness-browser.ts— provisions Kernel browsers via the SDKdirectly (
client.browsers.create/retrieve/deleteByID,captureScreenshot). Drops@onkernel/cua-translator'sbrowserSessionwrapper on new paths.src/harness-sessions.ts— wrapsJsonlSessionRepoforlist / find-latest / resolve-by-ref. Tolerant of unknown files in
the sessions root.
src/harness-models.ts— resolves-mthrough@onkernel/cua-ai'scatalog. Accepts
provider:modelrefs and bare ids that matchexactly one catalog entry; default
openai:gpt-5.5.src/harness-skills.ts— pi 0.79loadSkillsover~/.agents/skills/,<cwd>/.agents/skills/, and--skillpaths.src/harness-named-sessions.ts— named-session CLI re-implementedon the SDK. Metadata file format and path
(
$XDG_DATA_HOME/cua/named-sessions/<name>.json) preserved per theplan.
CLI surface
src/print.tsandsrc/action/harness-runner.tsdrive the harnessfor
--printand one-shot action subcommands(
open/click/type/press/observe/url/screenshot/do).Action runners count
turn_endevents and abort atmaxTurnsviaharness.abort.screenshotstays a direct SDK call.src/output/harness-jsonl.tssources the documented event schema(
session_created,browser_created,tool_call,tool_result,turn_done,assistant_text_done,run_complete,error, opt-indeltas + images) from
harness.subscribe. Adds aschema_versionfield on
session_created.src/cli-harness.tsorchestrates the new paths(
runPrintCommand,runActionCommand,runModelsSubcommand,runSessionSubcommand).src/cli.tsdispatchescua models,--print, action subcommands,and
cua session ...to the new wiring. Interactive entry pointcontinues on the legacy stack (intentional, per PR 1 scope).
--thinking <level>flag (defaultlow).Tests + CI
vitest.config.tsandtest/directory with:fixtures/scripted-provider.ts—registerApiProvider-baseddeclarative step DSL (text deltas / canonical CUA tool calls /
errors).
fixtures/fake-kernel.ts— plain object stubbingbrowsers.computer.{batch,captureScreenshot,getMousePosition,readClipboard},browsers.create/retrieve/deleteByID, andprofiles.{retrieve,create}.fixtures/harness.ts— assembles a realCuaAgentHarnessthroughharness.tsplus the scripted provider and fake Kernel.--printtext + jsonl envelope, action exit codes(
ok/not_found/screenshot), session resolution(list / latest / prefix / legacy-tolerance / ambiguity), model-ref
parsing (default / ref / bare-id / unknown / gemini alias), and
harness-assembly invariants (createCodingTools assignable to
extraTools, composed system prompt visible viabefore_agent_start, first-prompt screenshot delivered throughharness.prompt({ images }))..github/workflows/ci.ymlgains acli-unitjob that runs the newvitest suite on every PR.
Open questions called out in the plan
createCodingToolsis assignable to harnessextraTools. Confirmed: both useAgentToolfrom@earendil-works/pi-agent-core. Covered by theinstalls createCodingTools as extraTools by defaulttest intest/harness-assembly.test.ts.harness.prompt(text, { images }).Confirmed: the harness writes the user message with the image
content to the session on first turn. Covered by the
delivers the first prompt with an image attachedtest.JsonlSessionRepo.list.Confirmed:
list()skips unknown files and orphan directories inthe sessions root. Covered by the
tolerates legacy / unknown files in the sessions roottest intest/harness-sessions.test.ts. Per-cwd sessions remaindiscoverable; no
v2/subdirectory is needed.Behavioral contracts preserved
0ok,1not_found,2error / usage.formatCompact; errors to stderr.--print -o jsonlevent schema unchanged (only field sourcingchanges);
schema_versionadded.session start | stop | list | showoutput preserved.-c/-r/--session/--session-dir/
--no-sessionand named-session-s <name>continue to drivesession resolution.
--skill/-ns/--no-skillshonored on the new paths.Test plan
npm run typechecknpm test --workspace @onkernel/cua-cli(18 tests pass)npm test --workspace @onkernel/cua-agent -- --exclude "**/*.live.test.ts"(28 tests pass; no regressions)npm test --workspace @onkernel/cua-ai(88 tests pass)cua --printsmoke test against a live Kernel browser(deferred; CI exercises the same wiring against the scripted
provider)
Deviations from the plan
None. Interactive mode stays on the old stack as called out in the PR 1
section; PR 2 rebuilds it on pi-tui 0.79.
Out of scope (PR 2-4)
agent.ts, oldmodels.ts,config.ts, oldsessions.ts,skills.ts,agent-prompt.ts, and dropping the@mariozechner/*+cua-translator+cua-<provider>+smol-tomldeps (PR 3).packages/cua-{translator,openai,anthropic,gemini,tzafon,yutori}from the workspace (PR 4).
Note
Medium Risk
Medium risk from a large CLI execution-path swap (auth, sessions, Kernel browser lifecycle) while dual stacks coexist; mitigated by preserved contracts and new unit tests.
Overview
This PR moves non-interactive
cua(--print, action subcommands,models,session) ontoCuaAgentHarnesswith@onkernel/cua-agent,@onkernel/cua-ai, and the Kernel SDK—replacing the legacy agent/translator stack on those paths. The interactive TUI still uses the old wiring.New harness modules handle browser provisioning, JSONL sessions, model refs (
provider:model), skills, named sessions, print/jsonl output, and action runs (including turn caps viaharness.abortand SDK-onlyscreenshot).--thinkingis added; jsonl gainsschema_versiononsession_createdand provider-qualifiedmodelrefs.Vitest replaces node:test for CLI unit tests (scripted provider + fake Kernel fixtures), and CI adds a
cli-unitjob. Legacy@mariozechner/*deps remain for the TUI during the migration.Reviewed by Cursor Bugbot for commit c4d78e9. Bugbot is set up for automated code reviews on this repo. Configure here.